ar X iv : 1 10 3 . 08 90 v 2 [ cs . L G ] 4 M ay 2 01 3 EFFICIENT MULTI - TEMPLATE LEARNING FOR STRUCTURED PREDICTION

Conditional random field (CRF) and Structural Support Vector Machine (Structural SVM) are two state-of-the-art methods for structured prediction which captures the interdependencies among output variables. The success of these methods is attributed to the fact that their discriminative models are able to account for overlapping features on the whole input observations. These features are usually generated by applying a given set of templates on labeled data, but improper templates may lead to degraded performance. To alleviate this issue, in this paper, we propose a novel multiple template learning paradigm to learn structured prediction and the importance of each template simultaneously, so that hundreds of arbitrary templates could be added into the learning model without caution. This paradigm can be formulated as a special multiple kernel learning problem with exponential number of constraints. Then we introduce an efficient cutting plane algorithm to solve this problem in the primal, and its convergence is presented. We also evaluate the proposed learning paradigm on two widely-studied structured prediction tasks, i.e. sequence labeling and dependency parsing. Extensive experimental results show that the proposed method outperforms CRFs and Structural SVMs due to exploiting the importance of each template. Our complexity analysis and empirical results also show that our proposed method is more efficient than OnlineMKL on very sparse and high-dimensional data. We further extend this paradigm for structured prediction using generalized p-block norm regularization with p > 1, and experiments show competitive performances when p ∈ [1, 2). Structured prediction [18, 29, 33] has been successfully applied to the problems with strong interdependencies among the output variables. In the realm of Natural Language Processing (NLP), various tasks are formulated into structured prediction problems. A typical example is part-of-speech tagging which assigns a specific partof-speech tag to each token of an input sentence. The tag of one token is strongly correlated with the tags of its neighbors under the linear chain dependencies [18, 33]. More complicated structured output dependencies could be trees or graphs, such as Context-Free Grammar (CFG) [33], dependency parsing tree [23, 24], noun phrase coreference [13], and factor graph for relation extraction [40]. Note that there exist exact inference methods for sequences and trees. For the tasks with general output structures (eg., the pairwise fully connected undirected graph) the exact inference problem is intractable. In such cases, approximate inference is usually pursued to obtain an approximate solution [14]. The major advantage of structured prediction models such as Conditional Random Fields (CRFs) [18] and Structural Support Vector Machines (Structural SVMs) Qi Mao and Ivor W. Tsang are with School of Computer Engineering, Nanyang Technological University, Singapore 639798, e-mail {QMAO1,IvorTsang}@ntu.edu.sg.